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North Carolina’s Smart Start Initiative: 1994-1995 Annual Evaluation Report 

Summary 



Local Smart Start partnerships spent much of their first year (1993-94) in an intensive, 
comprehensive process of formulating their long-term plans to improve the lives of young children and 
their families. The formative evaluation of Smart Start focused on understanding this process in order to 
determine the strengths and weaknesses of the process as well as define a context in which to better 
understand partnerships’ implementations of their plans. The following points summarize the major 
findings of our evaluation of the first year process: 

• The hallmark of Smart Start has been its emphasis on local decision-making power, but the State has 
increasingly limited the degree of local autonomy through plan approval mechanisms, budgetary 
deadlines, and state-wide standards. Local partnerships intended to restructure local services from the 
bottom-up but have encountered significant obstacles directed from the top-down. 

• The focus of Smart Start has been very broad— covering early childhood education, family support, 
collaboration, and service systems change. With such breadth, local partnerships could not focus 
equally on all areas and, essentially, selected specific areas that were most appropriate for their 
community. This intentional diversity across counties may be narrowed considerably if local autonomy 
in program decision making is restrained by more State mandates. 

• Local planning is critical to Smart Start’s success. The first 12 pioneer partnerships needed more time 
to develop their long-term plans. Long-term planning involved bringing together a diverse group of 
community members to define the strengths and needs of their community, create a vision, select goals, 
and formulate strategies for reaching those goals. This long-term planning process was much more 
difficult when partnerships were forced to make immediate decisions at the same time. 

• Problems with the contracting system need to be solved. The first year efforts of the 12 pioneer 
partnerships were funded through 245 separate contracts. Although the contract approval time has 
shortened, the numerous steps in the contract approval process have not decreased significantly and 
continue to delay partnerships’ efforts. 

• The State provided technical assistance, through the County Collaboration project, for the process of 
working together to develop a plan. The County Collaboration process helped improve local 
partnership’s team functioning, and its meetings provided productive work sessions for partnerships. 
However, the State did not provide adequate content-specific technical assistance to the partnerships. 
Local partnerships needed more technical assistance regarding the content of their plans and strategies 
(e.g., which family support programs are most effective). 

• Involving parents in the decision-making process was difficult for all pioneer partnerships and requires 
continuing thought, attention, and action by both the State and local partnerships. 



Most local partnerships began implementing their plans in the spring and summer of 1994. Many 

partnership initiatives were designed to improve the quality of child care and provide additional family 

support services because these strategies have been shown to help children’s development. To date, 

significant changes are evident in the following areas: 

• Between 1993 and 1995, the percentage of AA-licensed child care facilities has increased much more 
in the pioneer Smart Start counties (25%) than in the non-Smart Start counties (17%). AA-licensed 
facilities generally have better staffxhild ratios, smaller group sizes, and more educational materials 
than A-licensed facilities. 

• Based on 193 child care visits, child care centers that received more Smart Start services were 
observed to provide higher quality care than centers that received fewer or no Smart Start services. 

• Child care centers that were participating in Smart Start efforts were more likely to have staff 
participating in T.E.A.C.H. (Teacher Education and Compensation Helps), a program designed to 
improve the education and compensation of child care teachers, compared to other randomly selected 
child care centers from the county/region. 

• Child care centers that were participating in Smart Start were more likely to offer children meals, 
speech and language screenings, developmental screenings, and dental screenings than other randomly 
selected child care centers from the county/region. 

• Currently, all 12 pioneer Smart Start counties/region provide Child Care Resource and Referral 
services to help families search for appropriate child care arrangements that meet their individual 
needs, whereas only 4 of the pioneer counties provided these services in 1993, prior to Smart Start. 

• Ten of the 12 pioneer Smart Start counties/region now have Family Resource Centers that support 
families by offering multiple services, support, information, and referral in a centralized location. In 
1993 prior to Smart Start, Family Resource Centers existed in only 3 of the pioneer counties. 

• Pioneer Smart Start counties effectively provided more child care through subsidies to children of low- 
income families. Pioneer Smart Start counties reduced the number of children on the child care 
subsidy waiting list by 42% from 1993 to 1994, compared to a 36% increase of children on the waiting 
list in non-pioneer Smart Start counties during the same time period. 



New, expanded, and improved programs in the child care, health, and family service systems have 
been implemented by the partnerships to enhance outcomes for children and families. After only a year, 
early childhood and family support services in the pioneer counties have improved in some ways, as 
described above. Continued improvement in these and other areas (e.g., child care teacher training, 
staffxhild ratios, family involvement) will be necessary to effect children and families’ lives significantly. 
Baseline data on children’s health status, the quality of early childhood education and care, family support, 
and service systems change have been gathered by the evaluation team and will be collected and monitored 
annually in the future to determine the long-term effectiveness of Smart Start. 
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North Carolina’s Smart Start Initiative: 
1994-1995 Annual Evaluation Report 



Smart Start is North Carolina’s partnership between state government and local leaders, 
service providers, and families to better serve children under 6 and their families. Smart Start’s 
innovative approach requires that local community partnerships plan how best to meet their own 
community’s needs, improve and expand previous programs for children and families, and design 
and implement new programs. In the fall of 1993 Smart Start established local partnerships in 12 
geographically and economically diverse areas across the state. Eleven of the partnerships 
encompassed individual counties and one was a seven-county cooperative effort. 

Following an intensive planning effort, these 12 partnerships were authorized to allocate 
from $355,000 to $2,334,000 for child and family services in 1993-94, with the amount partially 
determined by the number of county children under 6 living in poverty. In 1994-95 amounts 
ranged from $432,000 to $4,800,000 per site. Smart Start partnerships have put in, place a wide 
variety of activities to ensure that children are healthy and prepared to succeed when they enter 
school; that high quality, affordable early childhood education is available for every child who 
needs it; and that families are supported in their role as primary providers for their children. 

The evaluation of Smart Start is being conducted by faculty and staff at the University of 
North Carolina at Chapel Hill-researchers from the Frank Porter Graham Child Development 
Center and the Schools of Education, Public Health, and Social Work. In the first year of Smart 
Start, the evaluation team met regularly with staff from Department of Human Resources (DHR) 
and Division of Child Development (DCD) and with local partnership teams to determine the 
extent and type of evaluation that would be needed. We produced a comprehensive set of 
indicators for the potential use by all counties in their local evaluations and documented the first- 
year process of implementing Smart Start in a report, “Emerging Themes and Lessons Learned.” 

In the first year of Smart Start, evaluation team members also began collecting data from 
existing datasets in the child care, health, and family services areas. Whenever possible, we had 
hoped to use existing datasets, particularly those that are centralized in Raleigh on a county-by- 
county level, to monitor specific outcomes in the few years before Smart Start implementation 
and in each subsequent year of program implementation. This strategy would allow for 
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investigation of changes over time in Smart Start counties and comparison of participating and 
non-participating counties. However, we discovered severe limitations of most of these 
databases, including datasets collected on subsamples of children or families rather than all 
children or families; datasets collected by region rather than by county; datasets aggregated 
across wide age spans rather than by separate ages; and datasets collected by service counts 
rather than by counts of unique users. Therefore, most questions about Smart Start’s 
effectiveness must be answered by new data collection procedures developed by the evaluation 
team. 

Smart Start is a complex set of interventions, somewhat different in each county. Even 
within a county or region, Smart Start is not a program but rather several programs and projects. 
Some projects have well-defined participant lists, but others do not. Many Smart Start 
“participants” are not aware that they are receiving Smart Start supported services. For instance, 
a family may participate in activities at the local Family Resource Center and not even know that 
some of the services are funded by Smart Start. An evaluation of the overall Smart Start program 
must be broad and comprehensive. 

In September, 1 994, we completed a comprehensive evaluation plan that includes a 
variety of different data collection activities intended to capture the breadth of programs being 
implemented across the counties and the extent of possible changes that might result from Smart 
Start efforts (see Table 1 for a summary of the evaluation plan). This plan was circulated to and 
discussed with six of the local partnerships, DHR and DCD Smart Start core team members, the 
Legislative Oversight Committee, the North Carolina Partnership for Children, and three national 
early childhood and family support experts. With endorsement from these groups, the evaluation 
team has continued to implement the plan. 

An evaluation of Smart Start’s effectiveness requires long-term data collection. Ensuring 
that all children are healthy and prepared to succeed when they enter school, which is the primary 
goal of Smart Start, can be most reasonably measured after 5 years, when an entire group of 
children entering kindergarten would have had access to Smart Start services during their 
lifetimes. In addition, multiple, long-term, societal factors, such as poverty, affect children’s 
health and preparedness for school. Overcoming these problems requires changes in family 
supports and service systems, other long-term goals of Smart Start. In sum, North 
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Table 1: Summary of Smart Start Evaluation Plan: 
Goals 1 and 2 



Goal 


Expected Outcome 


Measurement Tool 


Source of Data 


1. Children are 
healthy and prepared 
to succeed in school 


more children will be 
prepared to succeed 
when they enter 
school 


Kindergarten Teacher Checklist 


kindergarten teachers 




FirstSTEP Screening Test 


kindergarten children 




local preschool screening 
measures 


public school records 


fewer children will be 
retained in the early 
school grades 


existing database 


SIMS database from NC DPI 


fewer kindergartners 
will be identified for 
the first time as 
having special needs 


existing database 


SIMS database from NC DPI 


higher scores on end- * 
of- year test in 3rd 
grade 


existing database 


SIMS database from NC DPI 


more kindergartners 
will be healthy when 
they enter school 


Kindergarten Health Assessment 


public school records 


health status of young 
children (b-5) will 
improve 


existing databases 


state-level health agencies (e.g., 
Medicaid, Health Services 
Information System) 


family interview 


families who use child care; 
families who use community 
services; 

families who participate in Smart 
Start efforts 


2. Families 
effectively fulfill their 
role as primary 
providers, nurturers, 
and teachers 


families value 
learning by 
participating in 
educational activities 
with their children, 
families have access 
to the support services 
they need and want, 
families act as role 
models through their 
involvement in the 
community, and 
families feel 
empowered 


family interview 


families who use child care; 
families who use community 
services; 

families who participate in Smart 
Start efforts 
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Table 1: Summary of Smart Start Evaluation Plan: 
Goals 3 and 4 



Goal 


Expected Outcome 


Measurement Tool 


Source of Data 


3. High quality, 
affordable services for 
children and families 


quality of child care 
will improve for all 
children, including 
those with special 
needs 


Early Childhood Environment 
Rating Scale, Infant/Toddler 
Environment Rating Scale, Family 
Daycare Rating Scale 


classroom observations of selected 
child care centers and family 
daycare homes 


regulatable aspects of 
child care will 
improve 


Child Care Director Interview and 
Family Daycare Home Provider 
Interview; Child Care Worksheets 
from Needs & Resources 
Assessment 


child care center directors and 
family daycare providers 


more child care 
providers will be 
well-trained 


Self-Assessment of Skills and 
Training Needs for Early 
Childhood Professionals 


child care providers in child care 
centers and family daycare homes 


more children with 
special needs will be 
served in inclusive 
settings 


Child Care Director Interview 


child care center directors 


existing database 


Part H and SIMS (Part B) 
databases 


service coordination 
for children with 
special needs will 
improve 


focus groups 


parents of children with 
disabilities and professionals who 
serve children with special needs 


more families of 
children with special 
needs will participate 
in and be satisfied 
with early 

intervention services 


family survey 


families of children with 
disabilities who are receiving early 
intervention services 


community services 
will be more readily 
available, affordable, 
and of high quality 


family interview 


families who use child care; 
families who use community 
services; 

families who participate in Smart 
Start efforts 


4. NC counties value 
children and families 
by providing options, 
resources, and 
encouraging 
collaboration 


inter-agency 
coordination and 
collaboration will 
improve over time 


Network Analysis Survey 


agency administrators and service 
providers 


more services will be 
available, affordable, 
and of higher quality 


family interview 


families who use community 
services and child care 


number of agencies 
that have parents on 
their advisory boards 
will increase 


agency survey; Child Care Center 
Director Interview 


agency administrators; child care 
center directors 
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Carolina cannot expect Smart Start to demonstrate major effects within a brief time span of a 
year or two. However, information collected during these first few years will establish a baseline 
from which to compare changes and can provide data regarding the direction of changes that may 
be emerging. Information from the first years of Smart Start is provided in this report. 

At the present time (May, 1995), some of the initial datasets are complete, but many 
others are in the process of being collected and will continue to be collected annually. To answer 
questions posed in this report, only verified data have been used. For example, 193 classrooms 
in child care centers have been observed and data from 169 of these visits are on file, checked, 
verified, and usable in data analyses at this time. For each type of data, this report will indicate 
what portion are being used in these preliminary analyses. When not enough data are available, 
we indicate that no conclusions can yet be drawn. These points bear repetition : this is an 
interim report using only data that have been verified to date, and any conclusions reached are 
tentative, subject to reanalysis with a complete dataset. This report does, however, serve as a 
blueprint of the types of data that will become available as data collection proceeds and the kinds 
of questions about Smart Start that can be answered. 

The report is organized in five major areas: 

• first year planning 

• early childhood education and care 

• family support services 

• health services 

• innovative service delivery/systems change 

Each section begins with a brief summary of why the topic is included in the evaluation and 
addresses several questions within each area. For each question, the methods for data collection, 
analyses, results, and interpretation are provided. 
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FIRST YEAR PLANNING 



A fundamental tenet of Smart Start is that local problems are best solved with local solutions. 
Partnerships spent a large portion of the first year discussing local needs and resources, 
developing goals, planning strategies, and beginning to implement activities. To assist 
partnerships with these tasks, the County Collaboration process was designed to ensure that 
partnerships build true collaborations that result in decisions made by a broad cross-section of 
community members. Each partnership created its own vision and plans, then supported many 
different projects. 

1. Was the County Collaboration helpful to Smart Start partnerships, and if so, in what 
ways? 

The County Collaboration improved team functioning as perceived by the partnership 
participants. Teams’ self-ratings in the areas of clarity of purpose, internal support, and external 
support increased positively over time. In other areas of team functioning such as cohesion, 
communication, and decision-making, initial team scores were quite high, indicating little need 
for improvement. 

Team functioning was measured by the Team Performance Scale (TPS), adapted from the 
Skills Inventory for Teams (Garland, Frank, Buck, & Seklemian, 1992). The TPS measures 12 
aspects of team functioning: clarity of purpose, cohesion, clarity of roles, communication, use of 
resources, decision making/problem solving, responsibility/ implementation, conflict resolution, 
view of family role, evaluation, external support, and internal support. Smart Start team 
members completed the TPS during the County Collaboration meetings in December of 1993, 
and February and April of 1994. Teams completed this rating a final time in August of 1994, 
approximately one year after their Smart Start collaboration began. 

Each team’s mean score on each item was examined over time. Over the four rating 
occasions, team members rated their team increasingly higher on purpose (e.g., mission), external 
support (e.g., support for the team from others), and internal support (e.g., procedures within the 
team that support the team’s work). Ratings in the other nine areas of team functioning did not 
show significant increases, probably because the initial ratings in December of 1993 were 
already at very high levels and, essentially, could not be improved. This could mean that teams 
selected to be in the first round of Smart Start were those who had an already-established 
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collaboration or those who, for whatever reason, cooperated well as a team when they first 
organized in the fall of 1993. 

According to interview data from 55 key Smart Start participants, these increased 
perceptions in team functioning were partially due to the support and guidance of the County 
Collaboration process. The majority of respondents thought that the County Collaboration 
meetings were very productive work sessions because teams were able to focus their attention 
and work collaboratively to accomplish pressing tasks. The majority of respondents (68%) also 
viewed the help of their county team facilitator (the “coach”) as important in their team-building 
efforts. Members from local partnerships strongly encourage the County Collaboration process 
to be maintained, although with some modifications (see the 1 995 Smart Start Evaluation Report, 
“Keeping the Vision in Front of You: Results from Smart Start Key Participant Interviews”). 

2. What important lessons were learned during the first year of planning? 

Several challenges that emerged during the first year of Smart Start were discussed in our 
1994 report, “Emerging Themes and Lessons Learned.” Some of the recommendations made in 
that report are summarized here: 

a) The focus of Smart Start, which was originally very broad and not clearly delineated, 
needed to be clarified. The local partnership leaders, the DHR core team, and the NC Partnership 
for Children developed a position statement, “Goals for Smart Start.” Although each individual 
goal is clear, there are 39 subgoals in all-far too many for a partnership to address. Essentially, 
each partnership selected specific subgoals that were most appropriate for their community. 

b) Local partnerships needed more time during the first year to formulate long-range 
plans instead of planning simultaneously for the short-term and long-term. The hallmark of 
Smart Start is its emphasis on locally developed plans that are responsive to local needs and 
resources. Bringing local leaders and families together to achieve consensus on a plan, usually a 
complex, multifaceted plan, required much time and personal effort. 

c) Problems with the DHR contract system needed to be solved. The contract problems 
prevented local partnerships from obtaining funds in a timely fashion, which in turn restricted 
partnerships’ abilities to implement activities during the first year. 
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d) The State and local partnerships needed to give parent involvement in decision- 
making more thought, attention, and action. Including nontraditional stakeholders as decision 
makers was difficult for most partnerships during the first year. 

e) Local partnerships needed content-specific technical assistance to help them select or 
design particular strategies that are most likely to help them reach their goals. 

3. What kinds of activities are funded through Smart Start? 

Seventy-three percent of Smart Start service allocations have been for early childhood 
education and care activities, 20% have been for family support (non child care related) services, 
and 7% have been for health services. Across the pioneer partnerships, proportions of dollars 
spent in the three areas vary. Within the early childhood education area, local partnerships’ 
allocations range from 55% to 91%. For family support services, allocations range from 6% to 
38%; and for health services, 2% to 1 8%. 

All 12 pioneer partnerships have implemented activities in these areas: child care subsidy 
expansion and enhancement, quality improvement grants to child care facilities, education and 
training for child care teachers, child care resource and referral services, as well as health visits 
or screenings. Ten have implemented activities to increase child care availability and have 
supported Family Resource Centers; 9 have funded transportation services. Several partnerships 
also began family literacy, parent education, and teen parent support programs. Some unique 
programs have been developed, including family support services for Hispanic/Guatemalan 
families, financial and educational support for parents who choose to remain at home to care for 
their young children, and service coordination for families who are homeless. 

Activities and allocations were obtained at the end of the third quarter from the Smart 
Start quarterly database that was designed by members of the Smart Start evaluation team and 
DCD staff. Activities were categorized into broad groups based on project descriptions and 
information from local partnerships. Local partnership administration and local evaluation costs 
were not included in the proportions above. 

4. How many new programs have been implemented because of Smart Start, and how 
many existing programs have been expanded because of Smart Start? 

Approximately half of the projects funded with Smart Start dollars during the 1994-95 
fiscal year are new projects created as part of the Smart Start initiative (151 out of 3 1 0 projects); 



the other half are expansions or enhancements of programs that existed prior to Smart Start (153 
out of 310). Six projects are continuations of existing programs, without expansion or 
enhancements. Smart Start is building on a base of existing community programs with 
expansion and improvement to some; Smart Start is also creating new programs when necessary. 
(Refer to question 3 for a description of the database from which this information was obtained). 

5. How many staff positions are funded with Smart Start dollars? 

This frequently asked question cannot be answered accurately. The Smart Start quarterly 
reporting system (see question 3 for a description) includes counts of staff funded directly with 
Smart Start moneys but does not include indirectly hired service delivery staff (e.g., a teacher 
hired with funds freed up because of Smart Start moneys). At the most recent reporting period, 
382 full-and part-time people (264 full-time equivalents) were directly employed by Smart Start, 
with a wide range reported across counties, even those of similar size. The variability is the 
result of the kinds of activities funded. For example, one county may directly employ several 
new teachers to staff Head Start and child care classrooms. Another county may increase the 
child care subsidy rate per child, with these funds used by child care directors to employ several 
new teachers. In both counties, new teachers were employed and more children were served, but 
the funding mechanism— and therefore the reporting system— differed. 

6. How many children and families have benefited from Smart Start? 

For several reasons, this question cannot be answered accurately. First, most service 
agencies and programs in North Carolina do not count individual families and children served 
but instead count service contacts (e.g., number of clinic visits) which overestimate the actual 
number of clients served. Second, some Smart Start activities are outreach efforts, such as public 
education through pamphlets, for which lists of families served cannot be produced. Third, 

North Carolina currently does not use a general database for all children and families served. 
Thus, people who receive multiple Smart Start services are counted by every different agency or 
program in which they participate. All of these problems prohibit the accurate calculation of a 
total number of children and families who have benefited from Smart Start. However, within 
some specific categories of services, such as the number of children removed from the child care 
subsidy waiting list, unduplicated counts are possible (refer to question 14 for child care subsidy 
waiting list information). 
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QUALITY OF CHILD CARE 

Research has indicated that children in higher quality child care demonstrate better cognitive 
and social skills than children in lower quality child care (Burchinal, Roberts, Nabors, & 

Bryant, in press; Cost, Quality, & Child Outcomes Study Team, 1995; White book, Howes, & 
Phillips, 1989). Because of this relationship between quality child care and better child 
outcomes, and because so many of North Carolina ’s young children are in child care, all of the 
partnerships are working to improve the quality of child care as a strategy for ensuring that 
children are prepared to succeed when they enter school. The evaluation includes measures of 
many components of child care quality. 

7. Are child care centers improving in quality as a result of Smart Start? 

In North Carolina an AA-licensed center has better staffxhild ratios, smaller group size, 
greater space per child and more educational materials than does an A-licensed center. The 
proportion of AA-licensed centers can be considered a marker for quality. Based on child care 
licensing data from the DCD, the percentage of AA child care facilities increased by 25% 
between 1993 and 1995 in the pioneer Smart Start counties, as compared to 17% in non-Smart 
Start counties. The new 1 994 ratio and group size requirements that affected all North Carolina 
child care programs (one of the state-wide elements of Smart Start) may have contributed to the 
large statewide increase in total numbers of AA facilities in only 2 years (20%). The higher rate 
of increase (25%) in AA facilities in Smart Start counties may be a result of local initiatives to 
support child care quality improvements. In fact, half of all the centers in North Carolina that 
have become AA-licensed since 1993 (62 of 120 AA licensed centers) are in the 18 pioneer 
Smart Start counties. The evaluation team is collecting observational data in over 175 child care 
centers that will also help answer this question in future years. 

8. What is the relationship between the quality of child care centers and Smart Start 
involvement? 

The extent to which a center is involved with Smart Start efforts and activities is 
significantly and positively related to the overall quality of care it provides. That is, centers that 
report receiving more Smart Start services are more likely to provide child care that is observed 
to be of better quality. The training, quality improvement grants, substitute pools, and all other 
local efforts funded by Smart Start may well be helping improve the quality of care in these 
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centers, but this analysis does not prove a cause-effect relationship. It is also possible that the 
differences in quality may have existed prior to Smart Start, with centers providing higher quality 
care more interested in participating in Smart Start. Data collected in future years will help 
clarify this relationship between Smart Start involvement and higher quality of care. 

Some center characteristics that are regulated by state licensing requirements are 
positively associated with quality (Whitebook et al., 1989). For instance, centers that have better 
staffichild ratios (i.e., more adults per child) and smaller group sizes tend to provide higher 
quality care. Higher quality has also been associated with better developmental outcomes for 
children. Staffichild ratios in infant classrooms in child care centers from the nominated sample 
(i.e., those indicated by the local partnerships as Smart Start participants) were significantly 
better compared to those in other randomly selected centers (1 :3.9 in nominated centers; 1 :4.8 in 
others). No significant differences in infant group size, toddler and preschool staffichild ratios, 
or toddler and preschool group sizes were evident between nominated and randomly selected, 
centers. 

These findings are from director interviews and child care center observations obtained 
from a sample of child care centers in each pioneer Smart Start site. In three pioneer counties 
with a small number of child care centers, all centers were asked to participate. In the remaining 
nine counties, two samples of child care centers were selected. One sample was a random 
sample of 10 or 20 licensed centers (20 in the larger areas-Cumberland, Mecklenburg, and 
Region A) selected from the DCD’s directory of all licensed centers. This sample is 
representative of child care in each Smart Start county/region. The second sample consisted of 
10 centers in each county /region that were nominated by the local partnership. This sample, 
known to have participated in the county’s Smart Start efforts in one or more ways, is included to 
study directly Smart Start’s effect on center-based child care. Of the 250 centers originally asked 
to participate, 1 93 (77%) agreed to be visited. 

Center involvement with Smart Start was measured through a question on the director 
interview. Directors were asked to indicate whether their center or staff had been involved in any 
of 14 different Smart Start supported activities aimed at improving the quality of child care, 
including teacher training, use of on-site educational consultants, use of a trained substitute pool, 
or purchase of new educational materials; each center received a score of 0-14. Every county 
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offered some, but not necessarily all, of these types of quality-improvement opportunities. 
Staffxhild ratios and group sizes were also obtained through questions on the director interview. 
Interview data from 129 child care center directors were included in these analyses. 

Quality of care was measured through on-site visits of preschool classrooms by trained 
observers using the Early Childhood Environment Rating Scale (ECERS) (Harms & Clifford, 
1980) Observational data from 169 child care centers were included in the analyses. The 
ECERS consists of 37 questions about the child care environment, activities, and teacher-child 
interactions. Each is rated on a 1-7 scale. Earlier research has shown that the total score and two 
factor scores, Appropriate Caregiving and Developmental Appropriateness, are the best measures 
of overall quality (Whitebook et al., 1989). The Appropriate Caregiving subscale includes items 
related to child-adult interactions and supervision. The Developmental Appropriateness subscale 
includes items related to daily schedule and educational materials and activities. Using data from 
the random sample of child care centers, the total ECERS score and both subscale scores were 
significantly, positively related to participation in Smart Start. 

9. Has the education level of child care providers improved? 

Child care providers with higher education levels are more likely to provide higher 
quality child care than providers with less education (Whitebook et al., 1989). Based on 
comparisons of data reported by child care directors in all the pioneer Smart Start counties/region 
from March 1994 to May 1995, the education level of child care providers in Smart Start sites 
has not improved significantly. However, child care centers in the nominated sample (i.e., those 
participating in local Smart Start efforts) were more likely to have teachers and directors who 
participated in the T.E.A.C.H. (Teacher Education and Compensation Helps) program that 
provides scholarship opportunities to improve the education of child care teachers while 
increasing their compensation. Increasing educational attainment, as a result of local partnership 
efforts or the statewide T.E.A.C.H. program, requires a good deal of effort to change and may be 
manifested over a longer period of time than the one-year period we have obtained. This is an 
indicator we will monitor each year. 

Baseline information on the education level of child care providers was obtained from 
364 child care facilities from the pioneer Smart Start counties in March, 1994, through the Needs 
and Resources Assessment of all 1 00 counties. In counties with fewer than 60 child care 
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centers/large homes, all child care centers/large homes were asked to provide data. In counties 
with more than 60 child care centers/large homes, a random sample of 60 centers/large homes 
was selected. These data showed that in the 12 pioneer sites, 36% of the providers had a Child 
Development Associate credential (CDA) or higher, and 17% had a bachelor’s degree or higher. 

Approximately one year into Smart Start implementation (spring, 1995), we gathered data 
on providers’ education level through interviews with child care directors in each pioneer Smart 
Start county/region. Data from 129 center directors and their teachers are available for analysis 
at this time (refer to question 8 for a description of the sample of child care centers). 

The teacher education data from the director interviews in the 12 pioneer counties/region 
showed that 6% of the providers had a CDA or higher, and 9% had a bachelor’s degree or higher. 
Analysis of variance statistical techniques show no significant improvements in teacher 
educational level from 1994 to 1995 and, in fact, showed a significant decrease in the proportion 
of providers with a CDA or higher. We suspect that differences in data collection techniques 
resulted in this anomaly and that a real decrease has not occurred. Each county was responsible 
for collecting its own data for the Needs and Resources Assessment, while data for the Smart 
Start evaluation were obtained through in-person director interviews conducted by trained data 
collectors. With the frequent teacher turnover rate in child care centers, it is also possible that the 
providers who were included in the 1994 sample were no longer working in child care centers in 
1995. 

10. Are Smart Start efforts improving child care teachers’ competence? 

The self-reported training needs and skills/knowledge of child care teachers in nominated 
centers were not significantly different from that of teachers in randomly selected centers, 
suggesting that Smart Start has not significantly changed teachers’ perceived competence. As 
with teachers’ level of education, changes with respect to professional competencies are likely to 
require intensive inservice training and may take longer than a year to achieve. 

Every child care teacher in the centers we observed was asked to complete the Self- 
Assessment for Child Care Professionals (Wesley & Buysse, 1994), which measures training 
needs and skills/knowledge across five competency areas: child development, classroom 
environment, curriculum and learning, professionalism, and children with special needs (refer to 
question 8 for a description of the child care sample). In general, child care providers rated 



themselves highest in the areas of arrangement of child care environments and professionalism; 
they rated themselves lowest in the area of serving young children with special needs. 

11. Have regulatable child care quality variables, in particular group size and staff:child 
ratios, improved over time? 

No significant differences in group size and staffichild ratios have been seen over the past 
year, although ratios have improved somewhat. The average preschool class included 14.5 
children in both 1994 and 1995; the mean staffchild ratio for preschool classrooms was 1:9.8 in 
1994, compared to 1 :9.4 in 1995. Using the partially completed dataset, we cannot conclude that 
these structural aspects of child care quality have significantly improved. Significant changes 
may not be revealed in only a year’s time or perhaps are aspects of quality that require more 
intense efforts than were received in the first year of Smart Start implementation. 

Information on ratios and group size was collected in the spring of 1994 from the Needs 
and Resources Assessment and compared to the recent data from the random sample of child care 
director interviews. (The Smart Start evaluation sample of child care centers was described in 
question 8 and the Needs and Resources Assessment sample was described in question 9.) 

12. Have services to children in child care centers increased as a result of Smart Start? 

This question is difficult to answer because no baseline data on the breadth or numbers of 
services to children are available. However, in the spring of 1995, child care centers known to be 
participating in Smart Start were more likely to offer children meals, speech and language 
screenings, developmental screenings, and dental screenings to children than other randomly 
selected centers. Information on the services offered by child care centers was obtained through 
director interviews (refer to question 8 for sampling and methods). We will annually monitor the 
number and kinds of services offered by child care centers. 
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FAMILY SUPPORT SERVICES 

One of the goals of Smart Start is to ensure that families effectively fulfill their role as primary 
providers, nurturers, and teachers of their children. Local partnerships are supporting families 
by offering a variety of educational opportunities and services to them. 

13. Are services for families increasing as a result of Smart Start? 

Child Care Resource and Referral (CCR&R) services help families identify high quality 
child care that meets their individual family needs. In 1993 prior to Smart Start, 4 of the 12 
pioneer counties/region offered CCR&R services; all Smart Start counties now offer CCR&R 
services. Every partnership funded (fully or partially) CCR&Rs during the 1994-95 fiscal year. 
Families in these counties have more help available as they search for appropriate child care 
arrangements. 

Family Resource Centers (FRC) support families by offering multiple services, support, 
information, and referral in a centralized location. In 1 993 prior to Smart Start, 3 of the pioneer 
counties/regions had FRCs; 10 now have FRCs and 1 more partnership is planning to establish an 
FRC. Eleven of the partnerships funded (fully or partially) FRCs during the 1994-95 fiscal year. 
Families in these counties have a source of childrearing help and support if they want it. 

Information regarding families’ use of and satisfaction with these services and child care, 
health care, and other community services is currently being obtained by the evaluation team 
through interviews with approximately 50 parents from each pioneer county/region who have 
participated in a Smart Start sponsored program. Data from these family interviews will be 
presented in a future report. 

14. Are low-income families benefiting from Smart Start? 

A portion of every pioneer Smart Start partnership’s funds are allocated to the 
government child care subsidy system that provides financial assistance for child care to low- 
income families. Data on the proportion of subsidized children in centers participating in Smart 
Start and data on the child care subsidy waiting list suggest that low-income families who use 
child care are benefiting from Smart Start. Based on director interview data, child care centers 
that received Smart Start funds for subsidies tended to enroll a significantly greater proportion of 
children who receive government child care subsidies based on income eligibility, compared to 
centers that did not receive Smart Start subsidy funds. 
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The proportions of subsidized children in centers were obtained from director interviews. 
Across the 12 pioneer sites, 35% of the children enrolled in an average center received child care 
subsidies. The range of subsidized children in any particular center was wide, from none of the 
children to all. (Refer to question 8 for a description of the sample of child care centers.) 

Additional information from the DCD subsidy database indicates that pioneer Smart Start 
counties reduced the number of children on the child care subsidy waiting list by 42% from 1993 
to 1994, compared to a 36% increase of children on the waiting list in non pioneer Smart Start 
counties during the same time period. Of the total number of children removed from the subsidy 
waiting list across all 100 North Carolina counties from 1993 to 1994, the pioneer Smart Start 
counties account for 56% of the reduction (1905 out of 3397) whereas the addition of children to 
the subsidy waiting list in pioneer Smart Start counties accounts for only 5% of the total addition 
across all counties (252 out of 5 1 1 1). 

15. Are children with special needs and their families receiving Smart Start services? 

In the first year of Smart Start, partnerships generally allocated less than 10% of their 
Smart Start dollars for activities specifically targeting children with special needs and their 
families. However, many of the other activities include children with special needs as part of a 
larger population of children and families to be served. This finding is not surprising, given the 
broad-based emphasis of the initiative on improving and expanding early childhood and family 
support services for all children and families. 

Smart Start projects that target or include children with special needs and their families 
were identified by reviewing the local partnerships’ first-year plans. Targeted activities focus 
exclusively on children with special needs. Activities that include children with special needs 
focus more broadly on all children and families but also explicitly mentioned children with 
special needs. We are currently repeating this analysis with partnerships’ second-year plans. 
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HEALTH SERVICES 

Because one primary outcome of Smart Start is to ensure that all children are healthy when they 
enter school, partnerships are implementing various strategies to improve the health of children, 
including special clinics and extended hours for immunizations and dental and developmental 
screenings, health consultants for child care programs, and mobile health vans. The diversity 
among partnerships in their health service initiatives is greater than in any other area. 

16. Are Smart Start efforts improving children’s health? 

Because no single standardized measure of children’s health status exists, health is 
measured through several different variables. Data regarding children’s health are being 
collected by the evaluation team and will be monitored annually through state databases, 
Kindergarten Health Assessments (KHA), and family interviews. Baseline data have been 
gathered, but at least two more years of data collection will be needed to answer this question 
adequately. Health is one area in which statewide databases may be of use to the evaluation, 
specifically databases for Health Check (formerly Early Periodic Screening, Diagnosis and 
Treatment, or EPSDT) data; health provider data; child abuse and neglect reports; Special 
Supplemental Food Program for Women, Infants and Children (WIC) participation rates; 
provision of services for children with special health needs (CSHN); and vital statistics data. At 
this time, these statewide data are only available for 1992-93 and/or 1993-94. Each county’s data 
are entered into the evaluation database to examine trends over time in these health variables in 
pioneer Smart Start counties compared to other counties. 

Data on the health status of entering kindergartners are being collected from the 
Kindergarten Health Assessments. The evaluation team cooperated with the State Center for 
Health and Environmental Statistics in a pilot test of the KHA and used the pilot data to select 
elements of the KHA as potential indicators of change in children’s health status. These data 
will be collected annually, monitoring trends over time. 

The final source of health data is being obtained through interviews with parents of 
children birth through age 5. (Refer to question 13 for a description of the family interview.) 
Specific health information is being gathered on children’s health status, utilization of health 
services for well child care and immunizations, injuries and illness, and use of referral services 
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for children with special health care needs, along with families’ satisfaction with and barriers to 
obtaining services. 

INNOVATIVE SERVICE DELIVERY AND SYSTEMS CHANGE 

A fundamental tenet of Smart Start is that local problems are best solved by local initiatives. 

With this assumption, the State committed itself to distribute funding to local programs 
according to recommendations from local partnerships. Collaboration among agencies and 
programs is also part of the State ’s vision of systems change through Smart Start. 

17. Has the State’s original intention of local autonomy been achieved? 

According to interviews with Smart Start participants and observations by Smart Start 
evaluation team members, the State has reined in the degree of local autonomy originally 
intended for Smart Start counties. Approval mechanisms with DHR and state budgetary 
deadlines forced partnerships to adopt more traditional approaches that did not necessarily match 
their ideals for service innovations. Certain state-wide standards became implemented in process 
and forced structural aspects to Smart Start initiatives that contradicted the premise of individual, 
local design. Also by implementing categorical standards and quotas, DHR restricted local 
partnerships’ leeway in designing new systems to serve families and children. In short, although 
local partnerships intended to re-structure local services from the bottom up, they have 
encountered significant obstacles directed from the top down. 

18. Are community agencies and programs working more collaboratively? 

One of the goals of Smart Start is the effective coordination of services for young 
children and their families, which requires agencies, both public and non-profit, to understand 
each other better and to collaborate on the planning and delivery of services. The existing parts 
of a system of service must be transformed into an integrated service system targeted at meeting 
the needs of young children and their families. Through questionnaires and interviews, a 
Network Analysis is being performed for each Smart Start county/region to determine the extent 
of community service providers’ awareness of other agencies providing services to families with 
children birth through 5 years, the degree of influence exercised by agencies, patterns of 
interaction between and among agencies, and patterns of referrals to and from agencies 
participating in Smart Start. Baseline data on these characteristics have been collected this year 
and will be compared to data in future years to determine subsequent changes in collaboration. 
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SUMMARY 



The long-term outcomes of Smart Start cannot yet be assessed, although some 
preliminary conclusions can be drawn about Smart Start’s implementation in the 12 pioneer 
partnerships. The challenge of bringing together an interdisciplinary team to craft local solutions 
to local problems has been met in some respects. Team functioning improved over the first year, 
thanks in part to the County Collaboration meetings, the work of the coaches, and the enormous 
personal effort of local Smart Start participants. The promise of local autonomy has not been 
achieved, however. Teams report being constrained by ever-increasing rules and procedures 
required by the State (at both agency and legislative levels). 

Several child and family services are more available now than before Smart Start: the 
number of child care resource and referral agencies and the number of family resource centers 
have increased dramatically. The proportion of AA-licensed centers in Smart Start counties has 
increased substantially. More children from poor or working-poor families are enrolled in child 
care through the subsidy program, and the subsidy waiting lists for child care have been reduced 
considerably. Teachers in child care centers that have been more involved with Smart Start 
efforts are more frequently working to improve their education through T.E.A.C.H. and are more 
likely to be offering higher quality programs. Staffchild ratios in infant classrooms are also 
better in centers participating in Smart Start. 

The findings mentioned above are some of the measurable, interim effects of Smart Start 
one year into implementation. The data supporting these findings, along with other data 
regarding early childhood education and care, children’s health, family support, and service 
systems change, need to be collected and monitored annually to determine significant changes. 
The long-term questions of Smart Start’s effectiveness for children and families will be answered 
in future reports. 
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